Microservices Observability and Monitoring with Prometheus, Grafana, and Loki

In a world of distributed microservices, visibility is everything.
Modern applications span dozens — even hundreds — of services across clusters, containers, and regions. Without proper observability, you’re flying blind when issues arise.

That’s where Prometheus, Grafana, and Loki come in — an open-source trio designed for metrics, visualization, and log aggregation. Together, they form the backbone of modern observability in cloud-native systems.

Why Observability Matters in Microservices

Monitoring tells you if something is wrong.
Observability tells you why it’s wrong.

In monolithic systems, debugging often meant checking a single log file.
In microservices, a single request might pass through API Gateway → Authentication → Payment → Notification — four different services with independent logs and metrics.

Without observability, finding the root cause could take hours (or days).
With observability, it’s seconds.

The Three Pillars of Observability

1. Metrics → Quantitative measurements of system performance.
2. Logs → Detailed event records for debugging.
3. Traces → Context of a request as it flows across services.

This article focuses on the first two pillars: metrics (Prometheus) and logs (Loki), visualized via Grafana.

The Open-Source Observability Stack

This stack gives you full visibility from data collection to visualization — and scales easily in Kubernetes or any cloud platform.

Architecture Overview

 ┌──────────────────────────────┐
 │        Application Pods       │
 │   (Microservices + Exporters) │
 └───────────────┬───────────────┘
                 │
          ┌──────▼──────┐
          │ Prometheus  │   ← (Collects metrics)
          └──────┬──────┘
                 │
          ┌──────▼──────┐
          │   Loki      │   ← (Collects logs)
          └──────┬──────┘
                 │
          ┌──────▼──────┐
          │  Grafana    │   ← (Visualizes data)
          └─────────────┘

Prometheus scrapes metrics, Loki gathers logs, and Grafana brings both together into a unified dashboard.

Step 1: Collect Metrics with Prometheus

Prometheus operates using a pull-based model, scraping metrics from services via HTTP endpoints.

Typical Setup

Each service exposes metrics at /metrics.

Prometheus scrapes these endpoints periodically.

Data is stored as time-series in its internal database.

Example: Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "microservices"
    static_configs:
      - targets:
        - "auth-service:9100"
        - "payment-service:9100"
        - "notification-service:9100"

Prometheus automatically collects CPU usage, memory, latency, and custom application metrics via exporters (e.g., Node Exporter, cAdvisor, Blackbox).

Step 2: Aggregate Logs with Loki

Traditional log management systems index every log line — making them slow and expensive at scale. Loki takes a different approach: it indexes only labels (metadata like service name, pod, or namespace) and stores logs efficiently in object storage.

Loki Advantages

• Seamless integration with Promtail or Fluent Bit for log shipping.

• Label-based querying (same syntax as Prometheus).

• Massive scalability with minimal resource overhead.

Promtail Configuration Example (promtail-config.yml)

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: "varlogs"
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: my-server
          __path__: /var/log/*.log

Logs collected by Promtail are sent to Loki, indexed, and ready to query directly inside Grafana.

Step 3: Visualize Metrics and Logs with Grafana

Grafana acts as the unified observability UI. It connects to Prometheus for metrics and Loki for logs — allowing you to visualize data side by side.

Add Prometheus & Loki as Data Sources

• Go to Grafana → Settings → Data Sources → Add Data Source

• Choose Prometheus and set URL → http://prometheus:9090

• Choose Loki and set URL → http://loki:3100

Example Dashboard Panels

• CPU Usage by Service

sum(rate(container_cpu_usage_seconds_total{job="microservices"}[1m])) by (service)

• Memory Usage Trend

sum(container_memory_usage_bytes{job="microservices"}) by (service)

• Error Log Trends (Loki Query)

{job="payment-service"} |= "ERROR"

You can correlate metrics and logs instantly — click on a graph spike to view logs from the same time window.

Step 4: Set Up Alerts and Notifications

Prometheus supports rule-based alerting, while Grafana can handle multi-channel notifications (Slack, Email, PagerDuty, etc.).

Example: Prometheus Alert Rule (alerts.yml)

groups:
  - name: service_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status="500"}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High 500 error rate detected"
          description: "Service has >5% error rate in the last 5 minutes."

Then configure Alertmanager or Grafana Alerting to deliver notifications to your chosen channels.

Observability in Kubernetes

Kubernetes environments are inherently dynamic — pods come and go. The Prometheus Operator and Loki Helm charts simplify deployment and scaling of this stack.

Example Architecture in Kubernetes

[Service Pods] → [Prometheus Operator] → [Grafana Dashboard]
                        ↓
                    [Loki + Promtail]

This architecture ensures:

• Automatic service discovery.

• Log collection from all pods.

• Real-time dashboard updates.

Benefits of Prometheus + Grafana + Loki Stack

Together, they empower DevOps teams to detect issues early, correlate signals, and restore service health faster.

Example Use Case: Payment Microservice Monitoring

Scenario:

The payment service occasionally returns 500 errors under heavy load.

Observability Flow:

• Prometheus shows spikes in request latency (http_request_duration_seconds).

• Grafana dashboard visualizes concurrent request load.

• Loki logs reveal database timeout errors.

• Root cause: DB connection pool saturation.

• Solution: Increase pool size and add caching.

In minutes, you’ve detected, diagnosed, and resolved a complex distributed issue.

Best Practices for Microservices Observability

• Use consistent labeling (service, namespace, env) for metrics and logs.
• Implement retention policies for old data.
• Use Grafana folders to organize dashboards per environment.
• Enable Prometheus remote write for long-term storage.
• Correlate metrics, logs, and traces for full-stack observability (add Tempo for tracing).
• Automate alerts and escalation workflows via Grafana Alerting.

Conclusion

In microservices environments, observability isn’t optional — it’s essential. With Prometheus tracking performance, Loki capturing logs, and Grafana tying it all together, you gain a 360° view of your system’s health.

This stack transforms your monitoring from reactive firefighting to proactive insight — enabling faster recovery, better performance, and happier users.